tokenize

Tokenize text strings for CLIP models

Overview

Converts text strings into token tensors suitable for CLIP text encoders. Uses the default SimpleTokenizer with BPE encoding.

Function Signature

def tokenize(
    texts: Union[str, List[str]], 
    context_length: int = 77
) -> torch.LongTensor

Parameters

texts · Union[str, List[str]] · required

Input text string or list of text strings to tokenize. Text is automatically cleaned and normalized.

context_length · int · default: "77"

Maximum sequence length for tokenization. Sequences longer than this are truncated. Default is 77 (standard for CLIP).

Returns

tokens · torch.LongTensor

2D tensor of token IDs with shape [batch_size, context_length]. Each sequence includes:

• Start-of-text token (position 0)
• Encoded text tokens
• End-of-text token
• Zero padding (if sequence is shorter than context_length)

Examples

Basic tokenization

import open_clip

# Tokenize single text
text = "a photo of a cat"
tokens = open_clip.tokenize(text)
print(tokens.shape)  # torch.Size([1, 77])
print(tokens[0, :10])  # First 10 tokens

Batch tokenization

# Tokenize multiple texts
texts = [
    "a photo of a cat",
    "a photo of a dog",
    "a photo of a bird"
]
tokens = open_clip.tokenize(texts)
print(tokens.shape)  # torch.Size([3, 77])

Custom context length

# Use longer context for more tokens
long_text = "a very detailed description with many words"
tokens = open_clip.tokenize(long_text, context_length=128)
print(tokens.shape)  # torch.Size([1, 128])

Complete inference example

import torch
import open_clip
from PIL import Image

# Load model
model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k')
model.eval()

# Prepare text
texts = ["a cat", "a dog", "a bird"]
text_tokens = open_clip.tokenize(texts)

# Encode text
with torch.no_grad():
    text_features = model.encode_text(text_tokens)
    text_features /= text_features.norm(dim=-1, keepdim=True)

print(text_features.shape)  # torch.Size([3, 512])

Handle long text with truncation

# Long text is automatically truncated
long_description = " ".join(["word"] * 100)
tokens = open_clip.tokenize(long_description, context_length=77)

# Check for truncation (last non-zero token should be EOT)
eot_token_id = 49407
print(f"Last token is EOT: {tokens[0, -1] == eot_token_id or tokens[0, tokens[0].nonzero()[-1]] == eot_token_id}")

Token Structure

Each tokenized sequence has the following structure:

[SOT] [token_1] [token_2] ... [token_n] [EOT] [PAD] [PAD] ...

• SOT: Start-of-text token (ID: 49406)
• EOT: End-of-text token (ID: 49407)
• PAD: Zero padding (ID: 0)

Text Preprocessing

The tokenizer automatically applies:

1. Basic cleaning: Fixes text encoding issues with ftfy
2. HTML unescaping: Decodes HTML entities
3. Whitespace normalization: Removes extra whitespace
4. Lowercasing: Converts text to lowercase (default behavior)

Notes

• This function uses a module-level SimpleTokenizer instance
• For custom tokenizers (HuggingFace, SigLIP), use get_tokenizer() instead
• Sequences longer than context_length are truncated, with EOT token placed at the last position
• Empty or very short texts still produce valid token sequences with SOT and EOT tokens

See Also

• decode() - Convert token IDs back to text
• get_tokenizer() - Get model-specific tokenizers